Storage system power usage

ABSTRACT

A method includes monitoring power usage for a storage system that includes a set storage units at a first level of storage granularity and a set of storage sub-units at a second level of storage granularity, wherein the second level of storage granularity is finer than the first level of storage granularity. The method further includes assigning a non-uniform power budget to the set of storage units and adjusting a power budget for the storage sub-units according to the non-uniform power budget assigned to the storage units. A corresponding computer program product and computer system are also disclosed herein.

BACKGROUND OF THE INVENTION

The present invention relates generally to managing power usage andspecifically to managing power usage in a storage system.

Without power management, data storage systems can intermittentlyconsume significant levels of power and create power usage spikes andsurges within a computing environment such as a data center. Forexample, date write operations and block erasure operations on flashstorage devices consume much higher levels of power (e.g., 10× and 30×respectively) than read operations. In response to the foregoing, avariety of power management approaches have been developed for storagesystems. For example, erasure operations associated with “garbagecollection” are often throttled within flash storage systems in order toreduce power consumption spikes and surges.

SUMMARY

A method includes monitoring power usage for a storage system thatincludes a set of storage units at a first level of storage granularityand a set of storage sub-units at a second level of storage granularity,wherein the second level of storage granularity is finer than the firstlevel of storage granularity. The method further includes assigning anon-uniform power budget to the plurality of storage units and adjustinga power budget for the storage sub-units according to the non-uniformpower budget assigned to the storage units.

A corresponding computer program product includes one or more computerreadable storage media and program instructions stored on the one ormore computer readable storage media, the program instructionscomprising instructions to monitor power usage for a storage system thatincludes a set of storage units at a first level of storage granularityand a set of storage sub-units at a second level of storage granularity,wherein the second level of storage granularity is finer than the firstlevel of storage granularity. The program instructions also includeinstructions to assign a non-uniform power budget to the plurality ofstorage units and adjust a power budget for the storage sub-unitsaccording to the non-uniform power budget assigned to the set of storageunits.

A corresponding computer system includes a storage system that includesa set of storage units at a first level of storage granularity and a setof storage sub-units at a second level of storage granularity, whereinthe second level of storage granularity is finer than the first level ofstorage granularity. The computer system also includes one or morecomputers and the above described computer program product.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is functional block diagram of depicting one example of a storagesystem in accordance with at least one embodiment disclosed herein;

FIG. 2 is flowchart depicting one example of a power distribution methodin accordance with at least one embodiment disclosed herein;

FIG. 3 is a dataflow diagram depicting one example of data interactionsrelated to power allocation in accordance with at least one embodimentdisclosed herein;

FIG. 4 is a table depicting one example of power budget allocation inaccordance with at least one embodiment disclosed herein; and

FIG. 5 is a block diagram depicting one example of a computing apparatus(e.g., computer) suitable for executing the methods disclosed herein.

DETAILED DESCRIPTION

As mentioned in the background section, flash storage systems oftendefer or throttle data write and erasure operations to prevent surges inconsumed power. However, wholesale deferral or throttling of suchoperations may adversely affect the performance of a data processingenvironment. Various embodiments of the present invention that addressat least some of the foregoing issues are described hereafter withreference to the Figures.

FIG. 1 is functional block diagram of depicting one example of a storagesystem 100 in accordance with at least one embodiment disclosed herein.As depicted, the storage system 100 includes a set of storage units 110comprised of storage sub-units 120, one or more storage controllers 130,and a power management module 140. The storage system 100 enables powerdistribution that is responsive to a variety of factors such as servicelevel agreements, policies, task priorities, and user priorities.

The storage units 110 and the storage sub-units 120 are physical orlogical units of storage that store data. In some embodiments, thestorage units 110 and/or sub-units 120 are physical units of storagethat are housed or packaged in separate physical domains. Examples ofsuch physical domains include sites, data centers, buildings, rooms,cabinets, enclosures, racks, shelfs, cards, devices, drives, circuitboards, and integrated circuits. The storage units 110 and/or sub-units120 may also be logical units of storage such as RAID groups, volumes(i.e., LUNs), and the like. Whether physical or logical, the storageunits 110 and sub-units 120 may comprise various types of storage mediaincluding flash memory, dynamic ram, rotational media, and tape media.

The storage controllers 130 control access to the storage units 110 andsub-units 120. The storage controllers 130 may be physically separatefrom, or integrated into, the storage units 110. For example, in someembodiments the storage system 100 is contained within one or more racksand each storage unit 110 and storage controller 130 corresponds to aphysically distinct space (e.g., shelf or piece of equipment) within theracks. Similarly, in some embodiments each storage sub-unit 120corresponds to a distinct drive (or volume) such as card-based flashdrive (or volume).

The power management module 140 manages the power budget for the storageunits 110 and sub-units 120. The power management module 140 may becentralized or distributed amongst the components of the storage system100.

FIG. 2 is flowchart depicting one example of a power distribution method200 in accordance with at least one embodiment disclosed herein. Asdepicted, the power distribution method 200 includes providing (210) astorage system comprising storage units and sub-units, monitoring (215)usage of the storage system, determining (220) factors relevant to powerbudget allocation, determining (230) a current power budget for eachstorage unit, determining (240) a current power budget for each storagesub-unit, and determining (250) whether to continue. The powerdistribution method 200 may be conducted by the power management module140 in order to control the distribution of power within a storagesystem such as the storage system 100.

Providing (210) a storage system comprising storage units and sub-unitsmay include providing a storage system such as the storage system 100depicted in FIG. 1. The storage system may be hierarchically structuredwith multiple levels of storage granularity. One level of storagegranularity may correspond to (i.e., managed as) storage units while afiner level of storage granularity may correspond to (i.e., managed as)storage sub-units.

Monitoring (215) usage may include monitoring the power usage (i.e.,consumption) of each of the storage sub-units as well as the storagesystem as a whole. In some embodiments, power usage is directly measured(e.g., via in-situ circuitry) while in other embodiments one or moremetrics that correlate to power usage, such as erasure rate and/or dataaccess rate, are used to estimate power usage.

Determining (220) factors relevant to power budget allocation mayinclude determining various configuration settings, performance metrics,and policies for a storage system that are relevant to intelligent powerbudget allocation. Examples of configuration settings and performancemetrics include power, throughput, bandwidth, storage topology andstorage budget/usage. The configurations settings, performance metrics,and policies may be managed and/or monitored at various levels ofstorage granularity such as a system level, a service level, a storageunit level, and a storage sub-unit level.

Determining (230) a current power budget for each storage unit mayinclude using the configuration settings, performance metrics, andpolicies for a storage system to determine a desired power budget foreach storage unit in a storage system. The power budget may be differentfor each storage unit due to the variation in usage, configurationsettings, performance metrics, and policies relevant to each storageunit.

Determining (240) a current power budget for each storage sub-unit mayinclude sub-allocating the power budget for a storage unit to thestorage sub-units that belong to the storage unit. In some embodiments,the power budget for a storage unit is evenly allocated to the storagesub-units that belong to that storage unit. In other embodiments, thepower budget for a storage unit is allocated to the storage sub-unitsbased on configuration settings, performance metrics, and policies foreach storage sub-unit.

In some embodiments, the current power budgets for the storage unitsand/or sub-units are hard limits. Applying those hard limits is referredto herein as power capping. For example, in response to determining acurrent power budget (i.e., power cap) for a sub-unit the current powerusage may be compared with the current power budget. If the currentpower usage for the sub-unit is above the power cap the sub-unit may bethrottled to bring the power usage in line with the power budget. If thecurrent power usage of the sub-unit is below the power cap, throttlingmay be reduce or eliminated for the sub-unit.

One of skill in the art will appreciate that throttling may beaccomplished in a variety of ways. For example, in some embodiments adevice erasure rate and data write rate may be limited (i.e., capped) toa level that corresponds to the current power budget. In otherembodiments, such rates are dynamically adjusted (relatively) higher andlower in response to and proportional to the difference between thecurrent power usage and the current power budget. In such an embodiment,the power budgets for the sub-units within a storage unit or the systemas a whole may be selected to provide a safety margin so that powerusage for some sub-units may drift temporally above their power budgetwhile maintaining a (statistical) power cap for the system as a whole.

Determining (250) whether to continue may include determining if ashutdown request or other terminating event has occurred. If aterminating event has not occurred, the method loops to the determiningoperation 220. Otherwise, the method ends.

FIG. 3 is a dataflow diagram 300 depicting one example of datacomputations related to power allocation in accordance with at least oneembodiment disclosed herein. As depicted, the dataflow diagram 300includes various input parameters (shown in rectangular boxes) andvarious calculated parameters (shown in rounded boxes). The inputparameters and calculated parameters are interconnected in a manner thatenables power distribution that is responsive to the various inputparameters. While a particular interconnection pattern is shown, one ofskill in the art will appreciate that a wide variety of interconnectionpatterns may be effective in determining power budgets for storage unitsand sub-units.

In the depicted example, a storage unit power budget 310 (e.g., for eachstorage unit) is calculated based on service level settings 314 andperformance metrics 314. Examples of (guaranteed) service level settingsand performance metrics include read operations per second, writeoperations per second, first byte read latency, and data throughput. Theservice level settings and performance metric may be selected/determinedon a LUN or RAID group basis. Subsequently, a sub-unit relative priority320 (e.g., for each storage sub-unit) is calculated based on storagetopology and mapping settings 322 and user scheduled priority 324.

In the depicted example, a sub-unit activity predictor or target 330 iscalculated based on system level performance metrics 332 and sub-unitperformance metrics 332. Subsequently an estimated sub-unit power 340 iscalculated based on the sub-unit activity predictor or target 330 andone or more sub-unit power measurements 342. Finally, a sub-unit powerbudget 350 is calculated based on system level power metrics 352, thesub-unit power measurements 342, and the estimated sub-unit power 340.

One of skill in the art will appreciate that the embodiments disclosedherein enable power allocation in a storage system that is dynamicallyresponsive to a wide variety of factors. For example, the embodimentsdisclosed herein are able to allocate storage unit and sub-unit powerbudgets that account for policies, assigned service levels, storagedemand under changing usage and system capabilities.

FIG. 4 is a table depicting one example of power budget allocation 400in accordance with at least one embodiment disclosed herein. Asdepicted, a raid enclosure 410 comprising an array of card-based flashdrives 420 is partitioned into a set of RAID groups 430 that each storeone or more volumes (LUNs) 440 thereon. A usage metric 440 and a servicelevel setting 450 are used to calculate a storage unit power budget 460for each RAID group 430 and storage sub-unit power budget 470 for eachflash drive 420. In response to determining the storage sub-unit powerbudgets 470, the flash drives 420 may throttle data access and/orerasure rates in order to adhere to the allocated power budget. In thedepicted embodiment, adherence to the sub-unit power budgets 470 alsoresults in adherence to the storage unit power budgets 460.

FIG. 5 is a block diagram depicting one example of a computing apparatus(e.g., computer) suitable for executing the methods disclosed herein. Itshould be appreciated that FIG. 5 provides only an illustration of oneembodiment and does not imply any limitations with regard to theenvironments in which different embodiments may be implemented. Manymodifications to the depicted environment may be made.

As depicted, the computer 500 includes communications fabric 502, whichprovides communications between computer processor(s) 505, memory 506,persistent storage 508, communications unit 512, and input/output (110)interface(s) 515. Communications fabric 502 can be implemented with anyarchitecture designed for passing data and/or control informationbetween processors (such as microprocessors, communications and networkprocessors, etc.), system memory, peripheral devices, and any otherhardware components within a system. For example, communications fabric502 can be implemented with one or more buses.

Memory 506 and persistent storage 508 are computer readable storagemedia. In the depicted embodiment, memory 506 includes random accessmemory (RAM) 516 and cache memory 518. In general, memory 506 caninclude any suitable volatile or non-volatile computer readable storagemedia.

One or more programs may be stored in persistent storage 508 forexecution by one or more of the respective computer processors 505 viaone or more memories of memory 506. The persistent storage 508 may be amagnetic hard disk drive, a solid state hard drive, a semiconductorstorage device, read-only memory (ROM), erasable programmable read-onlymemory (EPROM), flash memory, or any other computer readable storagemedia that is capable of storing program instructions or digitalinformation.

The media used by persistent storage 508 may also be removable. Forexample, a removable hard drive may be used for persistent storage 508.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer readable storage medium that is also part of persistent storage508.

Communications unit 512, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 512 includes one or more network interface cards.Communications unit 512 may provide communications through the use ofeither or both physical and wireless communications links.

I/O interface(s) 515 allows for input and output of data with otherdevices that may be connected to computer 500. For example, I/Ointerface 515 may provide a connection to external devices 520 such as akeyboard, keypad, a touch screen, and/or some other suitable inputdevice. External devices 520 can also include portable computer readablestorage media such as, for example, thumb drives, portable optical ormagnetic disks, and memory cards.

Software and data used to practice embodiments of the present inventioncan be stored on such portable computer readable storage media and canbe loaded onto persistent storage 508 via I/O interface(s) 515. I/Ointerface(s) 515 may also connect to a display 522. Display 522 providesa mechanism to display data to a user and may be, for example, acomputer monitor.

One of skill in the art will appreciate that the above disclosedembodiments may be adapted for a variety of environments andapplications. Furthermore, the programs described herein are identifiedbased upon the application for which they are implemented in a specificembodiment of the invention. However, it should be appreciated that anyparticular program nomenclature herein is used merely for convenience,and thus the invention should not be limited to use solely in anyspecific application identified and/or implied by such nomenclature.

The embodiments disclosed herein include a system, a method, and/or acomputer program product. The computer program product may include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry out themethods disclosed herein.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowcharts and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

It should be noted that this description is not intended to limit theinvention. On the contrary, the embodiments presented are intended tocover some of the alternatives, modifications, and equivalents, whichare included in the spirit and scope of the invention as defined by theappended claims. Further, in the detailed description of the disclosedembodiments, numerous specific details are set forth in order to providea comprehensive understanding of the claimed invention. However, oneskilled in the art would understand that various embodiments may bepracticed without such specific details.

Although the features and elements of the embodiments disclosed hereinare described in particular combinations, each feature or element can beused alone without the other features and elements of the embodiments orin various combinations with or without other features and elementsdisclosed herein.

This written description uses examples of the subject matter disclosedto enable any person skilled in the art to practice the same, includingmaking and using any devices or systems and performing any incorporatedmethods. The patentable scope of the subject matter is defined by theclaims, and may include other examples that occur to those skilled inthe art. Such other examples are intended to be within the scope of theclaims.

What is claimed is:
 1. A method comprising: monitoring power usage for astorage system comprising a plurality of storage units at a first levelof storage granularity, each storage unit thereof comprising a pluralityof storage sub-units at a second level of storage granularity, whereinthe second level of storage granularity is finer than the first level ofstorage granularity; assigning a non-uniform power budget to theplurality of storage units; and adjusting a power budget for theplurality of storage sub-units according to the non-uniform power budgetassigned to the plurality of storage units.
 2. The method of claim 1,further comprising throttling the plurality of storage units accordingto the non-uniform power budget.
 3. The method of claim 2, whereinthrottling the plurality of storage units comprises adjusting a flashdevice erasure rate.
 4. The method of claim 1, wherein the storagesub-units that belong to a particular storage unit are assigned asubstantially uniform power budget.
 5. The method of claim 4, furthercomprising throttling the storage sub-units that belong to theparticular storage unit according the substantially uniform powerbudget.
 6. The method of claim 1, wherein the non-uniform power budgetreflects a power related policy or service level.
 7. The method of claim1, wherein the plurality of storage units are logical devices.
 8. Acomputer program product comprising: one or more computer readablestorage media and program instructions stored on the one or morecomputer readable storage media, the program instructions comprisinginstructions to: monitor power usage for a storage system comprising aplurality of storage units at a first level of storage granularity, eachstorage unit thereof comprising a plurality of storage sub-units at asecond level of storage granularity, wherein the second level of storagegranularity is finer than the first level of storage granularity; assigna non-uniform power budget to the plurality of storage units; and adjusta power budget for the plurality of storage sub-units according to thenon-uniform power budget assigned to the plurality of storage units. 9.The computer program product of claim 8, further comprising throttlingthe plurality of storage units according to the non-uniform powerbudget.
 10. The computer program product of claim 9, wherein throttlingthe plurality of storage units comprises adjusting a flash deviceerasure rate.
 11. The computer program product of claim 8, wherein thestorage sub-units that belong to a particular storage unit are assigneda substantially uniform power budget.
 12. The computer program productof claim 11, further comprising throttling the storage sub-units thatbelong to the particular storage unit according the substantiallyuniform power budget.
 13. The computer program product of claim 8,wherein the non-uniform power budget reflects a power related policy orservice level.
 14. The computer program product of claim 8, wherein theplurality of storage units are logical devices.
 15. A computer systemcomprising: one or more computers; a storage system comprising aplurality of storage units at a first level of storage granularity, eachstorage unit thereof comprising a plurality of storage sub-units at asecond level of storage granularity, wherein the second level of storagegranularity is finer than the first level of storage granularity; one ormore computer readable storage media and program instructions stored onthe one or more computer readable storage media, the programinstructions comprising instructions to: monitor power usage for thestorage system; assign a non-uniform power budget to the plurality ofstorage units; and adjust a power budget for the plurality of storagesub-units according to the non-uniform power budget assigned to theplurality of storage units.
 16. The computer system of claim 15, furthercomprising throttling the plurality of storage units according to thenon-uniform power budget.
 17. The computer system of claim 16, whereinthrottling the plurality of storage units comprises adjusting a flashdevice erasure rate.
 18. The computer system of claim 15, wherein thestorage sub-units that belong to a particular storage unit are assigneda substantially uniform power budget.
 19. The computer system of claim18, further comprising throttling the storage sub-units that belong tothe particular storage unit according the substantially uniform powerbudget.
 20. The computer system of claim 15, wherein the non-uniformpower budget reflects a power related policy or service level.